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Originally,  the  determination  of  stationary  distributions  in  Markov  chains 
was  done,  using  completely  algebraic  arguments.  Unfortunately,  algebraic 
arguments  often  do  not  take  advantage  of  the  particular  structure  of  stochastic 
matrices  which  form  the  coefficients  of  the  equilibrium  equations.  Recently,  a 
new  trend  emerged  which  looks  into  the  probabilistic  interpretation  of  the 
algorithms  for  solving  equilibrium  equations.  The  objective  of  this  paper  is  to 
further  this  trend  by  deriving  computational  algorithms  from  purely 
probabilistic  arguments.  The  arguments  employed  in  this  paper  are  based  on 
the  semi-regenerative  structure  of  Markov  chains.  This  approach  gives  better 
insights  into  formal  manipulations  of  equilibrium  equations  and  provides 
probabilistic  interpretations  of  the  coefficients  obtained  at  each  step  of  such 
manipulations.  This  allows  one  in  turn  to  make  connections  and  draw 
conclusions  that  would  not  be  obvious  otherwise.  In  particular,  we  derives  a 
number  of  theorems  leading  to  a  better  understanding  of  Gaussian  eliminatian 


J 


a  semi-regenerative  processes.  These  theorems  allow  one  to  derive  relationships 
between  certain  matrix-geometric  solutions  and  Gaussian  elimination  Neuts 
[1981],  Gaver  et  al.  [1984]. 

Prom  a  Markov  chain  {Yn,  n=0,l,...}  one  can  obtain  a  semi-regenerative 
process  by  recording  the  state  of  the  chain  only  while  it  visits  points  of  a 
given  subset  D  of  the  state  space  of  Yn.  Specifically,  let  Tn,  n  s  1,2,...,  be  the 
time  of  the  n^  visit  to  D  and  let  Xn  be  the  position  of  the  chain  at  Tn.  The 
process  ((Xn,Tn),  n=l,2,..}  is  then  a  Markov  renewal  process  (see  Cinlar  [1975] 
Chapter  10),  (Xn>  n=l,2,...}  is  a  semi-regenerative  process  and  the  Tn  are  the 
semi-regenerative  epochs  of  the  Markov  renewal  process.  Thus,  if  we  consider 


An  =  {  .,Y, 


'n  "  1  'Tn-l* *  Tn_|+1  ******  T„-l^ 


then  the  conditional  distribution  of  the  sequence  (An,An+i,...),  given  the  past 
of  the  process  up  to  Tn_i,  depends  only  on  X„_i,  and  all  An  are  conditionally 
independent,  given  (X.,T.).  The  analysis  of  the  behaviour  of  Y.  from  one 
semi-regenerative  epoch  to  another  produces  the  main  relation  between 
steady-state  probabilities  that  is  used  for  developing  the  algorithm.  For 
simplicity,  we  assume  that  the  Markov  chain  [Yn,  n=0,l,..J  is  irreducible  and 
aperiodic,  and  that  the  state  space  of  Y.  is  E={0,1,2,...)  .  Such  a  Markov  chain 
reaches  steady  state  (see  Qinlar  [1975],  Chapter  8),  a  fact  that  is  expressed  by 
the  following  relation 
Yn  «*  Y. 

Alternatively,  one  can  write 

Pi(Yn  —  j )  pj  »  j  —  0,1,...  , 

Where  pj  is  the  distribution  of  Y,,  and  Pi(.}  ■  P{  .  IY()  =  i}.  The  notation  E( 
must  be  understood  in  a  similar  way. 


v,v.  V»  "*  •  ■%  ’VW\ 


D  =  |0,l,2,...,d-l) 


T  =  mini  m>0:  Ym  *  D). 


Let  li(Yn)  be  the  indicator  function  for  Yn=i,  and  let 

T 

vy(d)  =  Ej{  I  1  j(Ym)  }  =  El*{o»:  m  <  T  and  Ym  =  j  }.  (2) 

m=0 

The  symbol  *  denotes  the  cardinality  of  a  set.  The  expresion,  #{m:  m  >  T  and 
Ym  :  j  ),  in  particular,  is  the  number  of  times  m  meets  the  condition  in 
question,  and  the  quantity  vy(d)  becomes  the  expected  number  of  visits  to 
state  j  prior  to  the  exit  from  E-D,  given  one  starts  from  the  point  L  The  vj j  ( d ) 
are  related  to  the  pj  according  to  the  following  theorem: 

Theorem  1:  Let  (po,pi,..*)  be  the  steady-state  distribution  of  the  Markov  chain 


Yn.  Then 


d-1 

Pj  =  I  vij<d>  pi  . 
i=l 


Proof:  Let 


Ti  =  T 

Tn+l  =  mini  m>Tn:  Ym  *  D  } 

Xn  =  Y_  .  (4) 

Tn 

Prom  these  definitions,  it  follows  that  Tn  is  a  stopping  time,  and  that  Xn  can 
only  assume  the  values  0,l,...,d-l.  As  a  consequence,  the  strong  Markov 
property  applies,  which  means  that  Xn  is  a  Markov  chain  with  the  state  space 
D  =  |0,l,...,d-l).  This  chain  is  irreducible  because  the  original  chain  Yn  is 


irreducible.  Let  i  =  0,l,...,d-l  be  the  unique  invariant  distribution  for  Yn. 
Let  m(i)  =  E{{T}  be  the  times  between  regeneration,  and  let 
Knd.j)  =  Pi(Xn  =  j,  T>n} 

According  to  Qinlar  [1975],  Chapter  10,  Theorem  (6.12) 

lim  Pit  Yn  =  j  }  =  I  l  Kn(k,j)  /  l  m(k)V1  (5) 
n-*-  k*D  n=0  k*D 

We  must  mention  Qinlar  assumes  that  the  times  between  the  Markov-renewals 
have  a  continuous,  aperiodic  distribution.  Since  we  are  dealing  with  a 
discrete-time  Markov  renewal  process,  the  distributions  in  question  have  in 
fact  a  periodicity  of  1.  However,  the  proof  for  the  discrete  case  is  basically 
identical  to  its  continous  counterpart.  The  sum  of  the  Kn(k,j)  in  (5)  can  be 
found  as 

m  m 

l  Km(i,j)  =  I  Pit  Yn,  =  j,  T  >  m  } 
m=0  m=0 

m 

=  E  Ei(  lj(Ym)  lT>m  )  =  vij(d).  (6) 

m=0 

Since  Pi(  Yn  =  j  }  converges  to  pj,  one  finds  from  (5)  and  (6) 
d  d 

Pj  =  l  vy<d)  vi  /  I  m(k)i/k*  (7) 

i=0  k=0 

If  i,j  *  D,  vy^)  is  1  if  i=j,  and  zero  otherwise.  In  this  case,  (6)  becomes 

d 

Pi  =  vi  /  E  m(k)i/jt,  (8) 

k=0 

a  relation  that  is  also  well  known  from  semi-Markov  processes.  From  (7)  and 
(8),  one  easily  obtains  (3),  which  proves  the  theorem.  (3)  can  also  be  proven  in 
different  ways  (Miller  [1984],  Grassmann  et  al.  [1985]),  but  then,  the  connec- 
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tion  to  semi-regenerative  theory  is  lost. 

Later  on,  we  need  to  relate  the  vjj(k)  for  different  values  of  i,j,k.  Such 
relationships  can  be  obtained  from  the  following  theorem. 

Theorem  2:  Let  vjj(k)  be  given  by  (2),  and  let  pjj  be  the  transition 
probabilities  of  the  Markov  chain  Yn.  Then  for  k  >  d 

k-l 

vij(d)  =  vijW  +  Z  Viv(d)  vvjOt)  (9) 

v-d 

Proof:  Let  D  consist  of  the  first  d  points  form  0  to  d-1,  and  let  T  be  defined 
by  (1).  Similarly,  let  K  consist  of  the  first  k  points  from  0  to  k-l,  and  let 
t  =  min{m>0:  Ym«K)  (10) 

Since  the  first  visit  to  K  occurs  prior  to  the  first  visit  to  D,  r<T,  and  one  has 
vij(d)  =  Ej(*{m>0:  Y^j,  m<T)  ) 

=  Ei(t{m:  Ym=j,  0<m<r)  )  +  Ei(*(m:  Y„,=j,  r«m<T}  ) 

=  vijOO  +  Ei(«{m:  Ym=j,  r<m<T)  ).  (11). 

We  now  consider  the  successive  visits  to  K-D.  Thus,  let  ( (n)  the  n*’*1  visit  to 
K-D,  that  ia, 

f(l)  =  min(m:  m>0,  Ym  *  K-D) 
tin)  -  min(m:  m>f(n-l),  Yn  «  K-D). 

The  f(n)  are  obviously  stopping  times,  and  f(n)  •*  •  as  n  ■*  •.  Moreover, 
r  =  min(  m>0:  Ym*K  )  <  min(  m>0:  Ym«K-D  }  =  <(1).  (11) 

In  (10),  the  equality  holds  if  the  first  visit  to  K  is  also  a  visit  to  K-D.  In  a 
similar  way,  one  finds  from  (1)  and  (11),  provided  K3D 

r  <  T.  (12) 

(12)  holds  as  an  equalitiy  if  the  first  visit  to  D  coincides  with  the  first  visit  to 
K.  In  this  case,  the  second  term  of  (11)  is  zero.  If,  on  the  other  hand,  r<T, 


the  first  visit  to  K  precedes  the  first  visit  to  D,  that  is,  the  first  visit  to  K  is 
a  visit  to  K-D.  Hence, 


r=m)<T, 

and  one  finds  under  this  condition 


Bit  *{m:  Ym=j,  T<m<T)  ] 


-  Ej(  E  1  j(Ym)  lm<T  3 
m=f (1) 

- 

=  Bit  £  I  1  j(Ym)  lni<T  3 

v=l  n=f(v) 

•  f(v+l)-l  k-1 

=  Eit  E  I  lj(Ym)  I^t  I  l„<Yt(i))  ] 
v-\  m=f(v)  n=d 

k-1  -  f(v+l)-l 

=  Bit  E  t  I  E  lj(Y.)  1b<T  ln(Yf(l))  ]  ]  (13) 

n=d  v=l  ■=< (v) 


We  now  condition  on  Y{(„)  =  u.  Because  of  the  strong  Markov  property  for  the 
stopping  time  on  f(v),  one  has 


f(./+l)-l 

Et  [  E  lj(Ym)  lm<T  ^n^f{i/))  I  Y{(v)  =  u  ] 
m-S(v) 

f(l)-l 

=  Eit  E  lj(Ym)  lm<T  ln(Y0)  I  Y0  =  u  ] 
m-Q 

t-1 

-  En  [  E  lj(Ym)  ln=u  1  =  ln=u  vnj^ 
m=0 


We  now  use  (14)  to  write  (13)  as 


j-1  -  ((v+D-i 

Bit  I  I  l  lm<Yt<v)>  1*<„)<T  vnj<k>  1 

n-d  *=1  ta-f(v) 
k-1  • 

Bit  I  I  lm<T  Int^in)  vnj^k^ 

n=d  m=U 

k-1  T-l 

l  Bi  [  I  vnl(k)  in(Ym)  ] 

n=d  m=0 


I  vnj00  vin(<D. 
n=d 


In  summary 


Bi[  t(m:  Ym=j,  r<m<T}  ]  =  JL^in^^nj^  (15) 


If  we  compare  (15)  with  (II),. we  obtain  (3). 
Corollary:  For  i<d 


vijt1*)  =  v.j(d+l)  +  vdjtd+l).  (16) 

A  similar  result  was  obtained  in  Grassmann  et  al  [1983]. 

Applications 

The  theorems  above  are  particularly  useful  if  the  states  can  be  partitioned 
into  groups  of  states  Ko,Ki,K2i...,  called  levels.  Level  0  consists  of  state  0  only, 
and  level  n,  n=0,l,2,...  consists  of  the  states  kn.[  to  kn-l.  For  consistency,  ko 
is  defined  as  0.  If  d  in  equation  (3)  is  replaced  by  k<j,  one  has 


J«Kd 


(17) 
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kd-l  (kd)  d-1  t*d) 

Pj  =  I  Pi  vij  =  I  I  Pi  vy 
i=0  n=0  i*Kn 

We  now  define  the  following  vectors  and  matrices: 
dn  =  t  Pj»  J*Kn  1 


and 


„  .  r  (kd)  .  .  _  . 

Und  -  t  vy  ii*Kn»  j*Kd  1 

Using  these  symbols,  (17)  becomes 

d-l 

dd  =  E  dn  Und 
n=0 


(18) 


(19) 


In  many  applications,  one  has  the  following:  For  all  d  within  a  certain  range, 
say  d>c,  level  d  cannot  be  reached  from  level  n,  n<d-l.  In  other  words,  level 
d-l  cannot  be  skipped.  Such  levels  will  be  called  non-skippable.  If  d  is 


non-skippable,  Ud-2»  Ud-3»..«  are  all  zero,  because  all  vjj^d)  vanish.  In  this 
case,  (19)  becomes 

dd  =  dd-1  Ud-Ifd  (2  ) 

Suppose  now  that  for  n  within  the  range  extending  from  a  certain  lower  limit  e 
up  to  •,  ail  levels  have  the  following  properties: 

1)  Level  d  is  non-skippable 

2)  kd+i  =  kd  ♦  a,  where  a  is  a  constant 

3)  [pi  j+i,  i*Kdl  -  [pi  j+i»  i*Kg]»  j*0,  g>d. 

It  is  easy  to  verify  that  under  these  conditions,  Ud-lfd  =  =  U,  g>d. 

From  (20) .  one  concludes  that  under  these  conditions,  qd  is  matrix-geometric 
(Wallace  [1969],  Neuts  [1981],  Gaver  et  al.  [1984]),  that  is 

dd+r  =  dd  U*\  (21) 
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Thus,  using  the  v|j(d)  allows  one  to  derive  a  number  of  results.  Moreover, 
the  vy(d)  can  be  found  in  many  cases  of  practical  importance.  Grassmann  et  al 
[1985]  showed  how  to  find  the  vyO),  using  a  modification  of  Gaussian 
elimination.  Once  the  vfj(j)  are  determined,  one  can  obtain  the  vij(d>»  d<j, 
either  using  (9)  or  (16).  Moreover,  the  vjj(d)  can  be  grouped  into  matrices 
according  to  equation  (18),  which  allows  one  to  set  up  interesting  relations  for 
quasi  birth-death  processes,  in  particular  equations  (20)  and  (21)  . 
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